The interest in wearable technology has grown considerably over the last decade. For example, augmented reality (AR) displays may be worn by a user to present the user with a synthetic image overlaying a direct view of the environment. In addition, wearable virtual reality (VR) displays present virtual images to the user to provide a virtual environment. When representing virtual elements in a three-dimensional virtual environment, such as a VR environment or an AR and/or mixed reality environment, it is often difficult to provide for interaction between the virtual element and a user who exists in the real world. For example, a user may wish to select, move, translate, rotate, stretch, compress, deform, or otherwise interact with a virtual element. However, providing such interaction is often difficult.
Some AR and VR display systems are gesture based and attempt to recognize user gestures in order to allow a user to interact with the virtual environment. For example, sensors attempt to recognize movements of the user, one or more positions of user body parts, or specific user configurations, such as hand gestures (e.g., a thumbs up gesture). However, gesture based systems have a number of drawbacks. For example, a gesture based system may not be able to quickly recognize a gesture, may determine a different, unintended gesture, or may not be able to recognized a gesture at all which lead to user frustration and a bad user experience. In addition, the user may be required to “learn” the gestures, some of which may not be recognizable or easy to produce for all users. Moreover, in a global economy, some gestures may not translate across geographic or cultural boundaries resulting in having to add or build in different gestures for different users.
In addition, some systems require the user to use additional equipment, such as a wand, wristband, makings, or other additional equipment. Such solutions add expense, may be lost, or make the system less portable all of which add to inconvenience for the user.
Some AR and VR display systems also have difficulty determining whether a user is interacting with a virtual object. For example, the systems require a large number of sensors or external sensors with one or more fiducial markers to attempt to accurately map a user's environment, and then accurately map the user's position in the real world to the virtual environment. Such system may not be able to accurately identify which virtual object that a user is attempting to interact with, requiring the user to verify or retry the desired interaction, or worse incorrectly identify another object that is not desired. Such systems also may be computationally complex resulting in delay in determining and displaying the interaction, all of which leading to an experience that may not be realistic or credible to the user.